[None][feat] Add PrimTS MLA prefill, MTP, and Rubin support by yuxianq · Pull Request #19395 · NVIDIA/TensorRT-LLM

yuxianq · 2026-09-18T04:43:06Z

Description

Integrate expanded MLA prefill into PrimsTSFmha, so Kimi K3 full-attention layers can use PrimTS for context and decode with BF16 or FP8 target KV cache. Support fresh, cached-prefix and chunked context, linear MTP, and Helix per-query local KV bounds. KDA execution and the default FMHA library order are unchanged; dense PrimTS remains opt-in.

This revision rebases onto TRT-LLM main a1c22fc7a5d09f65afcc66313f85d680d20afb98, including the PrimTS refresh in #21354. The temporary FlashInfer source branch is now based on canonical trtllm-prims-ts-dev-20261107 at 911d6a9d62308b6c8a5249e8dc4aabc7518020f1; the canonical branch itself is untouched.

Reuse native MLA RoPE/cache update and FP8 preparation. Expanded context supports QK/V dimensions (128,128), (192,128), and (256,256), with the existing native latent512/RoPE64 integration.
Pass aligned row-strided FP16/BF16 V directly to the context kernel. Remove its packing copy and PrimTS-specific reservation; FP8 conversion buffers remain.
Export optional natural-log maximum/denominator statistics for partition merging, independent of LSE. Preserve linear-MTP/Helix bounds, empty local partitions and CUDA-graph replay.
Adapt wrapper and warmup calls to the refreshed compile/run APIs. Plan/run scale argument kinds match, original natural FP32 scales are retained, and compact split workspace indexing follows the actual split count.
Retain main's dependency-based availability checks and standard FP8 KV support while preserving this PR's SM107 MLA eligibility. The dependency pins CUTLASS DSL4.8.0; compiler checks remain.
Keep the strict PrimTS K3 FP8/DSpark end-to-end CI entry and targeted module, native-preprocessing, cache-reuse, merge and graph coverage.

Review order

Native context preparation and its integration: 3d7543ea79, a0c1277248.
Rubin, linear MTP, Helix bounds and K3 coverage: dac0ce3750 through d6978a1f8e.
Consumer performance changes: capacity normalization 1ab5362bc5, startup profile warmup ba5dd568f0, workspace-size caching 6a7e605776.
Prepared-launch ABI/fixtures and original-scale correctness, then reviewed geometry and direct-V changes: 6137dedb02 through f7a0200f37.
Refreshed source import 3aa9939269, adapter/warmup API reconciliation d745d03ae1, and GPU test consolidation 6df6036df4.

The pinned FlashInfer history retains individual functionality, kernel-performance, correctness, strided-V and rebase-fix commits. Obsolete vendor-only snapshots were replaced by the audited import of that rebased series. Historical functionality/performance checkpoints below remain historical; their measurements do not qualify this refreshed head.

Vendored source

Temporary source branch: yuxianq:prims-ts-mla-prefill--flashinfer.
Immutable pin: 26b197614300b549eb103fc59d736c3a0e4b3fa9. Corresponding upstream work: FlashInfer #5489, synchronized to the same head.

Vendor state is managed with scripts/vendor/manage.py. Preserve current main's import isolation, precompile/persistent-cache machinery and existing adaptations. The regenerated compatibility patch removes a redundant copyright hunk and forwards the new optional statistics/bounds slots from the TRT-only prepared sparse launch. Kernel fixes remain upstream source commits, not hidden compatibility patches. Other vendor entries are unchanged.

schema_version: 1
vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/5489

Latest main rebase validation

Rebased 06609107ac to 6df6036df4 on main a1c22fc7a5. All 58 changed files remain. The package-version-removal commit is already in main, so 21 commits become 20 without losing functionality or performance work.
Resolved overlaps in PrimTS admission/workspace logic and tests: retain standard FP8 preprocessing/device scales, MLA context/MTP/Helix, and Rubin MLA/P128 routing. The old Blackwell-only test restriction is redundant with this PR's existing SM107 support.
FlashInfer source pin, compatibility patch and vendored tree are unchanged byte-for-byte from the previous PR head.
Fresh matching native B200 build passed. The complete 567-case selection passed with 0 failed / 0 skipped, covering standard FP8 and native GQA, expanded MLA geometries/strided V, K3 cached/chunked context and FP8 MTP5/MTP7, scale/device-metadata graph replay, prepared sparse launches, page-size routing, and phase warmup. This is component validation, not a new full-model or multi-GPU run.
The first run passed all 567 test assertions but exited 4 because pytest's import-path guard rejected the temporary runner directory automatically added by Python. Correcting only the local launcher to python -P gave a complete 567-pass rerun with exit 0, with the guard still enabled. No product-code or test change was needed; reruns are not additional unique coverage.
Full pre-commit, syntax/undefined-name checks, remote Pyrefly, and vendor integrity checks passed. Installed native/Python origins and all 58 changed-file hashes match the tested source. The prior test consolidation is retained; no new parameter grids or CI entries were added by this rebase.

Previous source-refresh validation

Fresh B200 native build completed with CUDA13.4, Torch2.14.0a0, CUTLASS DSL4.8.0 and FlashInfer0.7.2. The local out-of-tree attribution scanner warned about its working directory; the wheel is a validation artifact, not a release build.
The matching-wheel 522-case selection first reported 514 passed / 8 failed / 0 skipped. All failures were the prepared sparse MLA ABI adaptation and its old-schema fixtures. After appending the two optional slots and updating the fixtures, all 8 affected checks passed, including graph replay, one/two-pool paths and persistent-kernel reload. This is a full run plus a targeted post-fix rerun, not a second full 522-case execution.
All selected K3 full-attention component cases (BF16/FP8 KV, cached/chunked context, TP-local/DP head geometry), FP8 linear MTP, expanded geometries, FP8 scales, and dense graph replay passed. This is component qualification, not a new full-model or multi-GPU K3 run.
Native library/module origins and changed-source hashes were verified against the installed wheel. Final committed files exactly match the post-fix qualification manifest. Full pre-commit checks and remote Pyrefly passed; vendor integrity checks passed against local and published source.
FlashInfer's full selected source run reported 525 passed / 8 hardware-policy skips / 2 failed. The rebase-induced device-scale reducer compiler error was fixed; 22 targeted rerun cases passed. The remaining BF16 Helix boundary tolerance failure reproduces identically on both old source branches (a4b6e9028 and 64273c074): 2/884736 elements, greatest absolute error0.0006412482 versus atol0.0005. Its test, reference and tolerances remain unchanged. The 22 cases overlap the full run and are not additional unique coverage.
Before/after patch-range and changed-file audits preserve the previous integration and performance work. The old-dev-only sink helper is superseded by the refreshed implementation and covered by strict sink-scale regressions.
Test consolidation preserves distinct geometry/dtype/layout/topology contracts and the required K3 CI case. TRT native layout cases: 48→20; cache integration: 36→18; independent capacity boundaries: 144→18. FlashInfer affected-file collection: 2391→2373, including new device-scale/compact-split regressions. No proportional CI speedup is claimed.
No new end-to-end or performance experiment was run for this rebase. FlashInfer CI remains user-managed; no bot command is issued there.
Keep the TRT-LLM PR in draft pending current-head CI and review. Multi-GPU CI approval must follow the normal maintainer policy.

Historical performance and accuracy — unchanged, not refreshed-head qualification

Main-based prepared-context comparison on Rubin

After the correctness gates passed, 36 fresh-process receipts completed three ABBA blocks per case. Baseline e2203f939e6c95475d7ec609fadfb916c04a2bec and candidate 005d91701b95e88a3495da84e6d384efccc606d7 contain the same original-scale correctness change. Their production delta is only full-tile dense LDRED plus request-local dense KV domains. Both use actual runtime Q/K maxima.

Synthetic case	Baseline graph latency	Optimized graph latency	Mean paired change	Sample SD
One request, statistics ON	100.212 us	91.859 us	-8.335%	0.089 pp
Two ragged/empty requests, statistics ON	376.195 us	102.620 us	-72.721%	0.018 pp
One request, statistics OFF	73.045 us	72.087 us	-1.312%	0.040 pp

All cases use packed dense FP8 E4M3 attention, H96, Dqk192/Dv128, and BF16 output. The one-request cases use Q/K lengths 1920/1920 and Q/K capacities 2048/2048. The two-request case uses Q lengths [1664,256], K lengths [0,3840], and capacities Q2048/K4096. All eager, graph, and post-timing numerical checks passed; input identities were unchanged. An independent audit recomputed all 360 raw CUDA-event samples, receipt medians, and nine block summaries. The means of paired block ratios are reported, with sample SD across three blocks.

This is one-allocation prepared-wrapper graph replay, not a CuTeDSL/trtllm-gen comparison or a new K3 end-to-end result. The large empty-partition gain does not establish how frequently that geometry occurs in serving. Statistics ON/OFF also changes precision policy, so their absolute-time difference is not an isolated statistics-store cost.

Functionality-only Kimi K3 qualification on Rubin

365 checks passed, including real K3 MLA module coverage with MTP5/MTP7 and backend/selection contracts.
A 1,319-question GSM8K run scored 96.058% with strict PrimTS, FP8 target KV/P64, TP8/EP8, attention DP disabled, DSpark7, CUDA graphs, and FP32 KDA state. No forced KDA-engine override was used in this functionality run.
This validates the functionality-only snapshot (26e9b3b33c plus FlashInfer d9f0146; published source-equivalent functionality commit df3434b1d), not every current-main combination.

Controlled DEP16 performance experiment on Rubin

The performance experiment used FP8 target KV/P128, TP16/EP16 with attention DP, DSpark5 (six target query tokens per verification step), chunked prefill, overlap scheduling, and CUDA graphs. Both policies used the same BF16 KDA state and engine prefill path; drafter KV remained BF16/P32. The fixed workload was 4,096 input / 256 output tokens at concurrency 128 and 256, seed 42, temperature 0, and ignore-EOS. This is an aggregated-serving comparison, not the original disaggregated AgentX workload.

Three allocation-bounded ABBA blocks produced 24 accepted measured windows. All measurements were retained; sealed-source/artifact and compilation-log audits passed. Observed native compilation warnings were confined to startup; no matching compilation markers occurred in accepted measured windows. This does not prove zero unlogged JIT.

Concurrency	CuTeDSL + trtllm-gen output tokens/s	Optimized PrimTS output tokens/s	Mean paired-block change	Sample SD
128	3,077.84	3,144.82	+2.18%	2.57 percentage points
256	4,671.57	4,696.30	+0.56%	2.03 percentage points

This supports near throughput parity in this workload, not a robust speedup, statistical equivalence, or noninferiority claim. Policy TPS columns are arithmetic means; percentage changes are means of paired block ratios. The experiment used the pre-rebase TRT be927b7015 runtime and the source-equivalent FlashInfer 30e7ea65/76e822974 kernels. It is not a latest-main native performance result or a causal before/after comparison across different allocation series.

Accuracy and scope limits

For the DEP16 performance configuration, native GSM8K scored 96.5125% (1,273/1,319) and PrimTS 95.8302% (1,264/1,319). Both passed the reused 93.2975% gate, but there is no dedicated FP8+DSpark reference row. The nine-answer net gap and exact-text differences remain; output equivalence and deployment suitability are not established.
Historical CP8 prepared-attention plus real FIFO2 component measurements showed approximately 1.09–1.10x graph speedup for their static-capacity/single-split composition. They are not full-model Helix results or a rerun of the final kernel bundle.
Strict 8-GPU aggregated full-model Helix is unsupported by the current scheduler/model and was excluded from the revised validation scope.
The benchmark's shared KDA-engine choice does not fix the separate Small-BH NaN issue, and no KDA behavior change is shipped here.
Keep this PR in draft pending current-head CI and human review.

PR Checklist

Changes follow TRT-LLM coding guidelines and DCO requirements.
No public LLM configuration fields or model identifiers are added.
Vendor/dependency CODEOWNER requirements remain applicable.
Component, full-checkpoint, historical-performance and current-head evidence are distinguished.
PR scope, motivation and limitations have been reviewed.

GitHub Bot Help

To see available CI commands, comment /bot help.

yuxianq · 2026-10-09T04:14:40Z

@yuxianq — vendor promotion needs author action:

The pinned temporary source commit e377847b798c8203eec83802ea81794bf07303f1 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto ae3851ab5f958ba22f154059df5b38321f6369d2. Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

yuxianq · 2026-10-09T08:17:24Z

/bot run

tensorrt-cicd · 2026-10-09T08:24:02Z

PR_Github #76928 [ run ] triggered by Bot. Commit: d575c3f Link to invocation

tensorrt-cicd · 2026-10-09T12:14:05Z

PR_Github #76928 [ run ] completed with state SUCCESS. Commit: d575c3f
/LLM/main/L0_MergeRequest_PR pipeline #63420 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-10-13T08:23:51Z

@yuxianq — the PrimTS upstream refresh #20155 has merged. Please update this source-update PR before merging it.

The new canonical source branch is yuxianq/flashinfer:trtllm-prims-ts-dev-20261012, initially pinned at ae312f7cc1723b06fda5168105ade0f328ee47d8.

Rebase your temporary FlashInfer branch onto the new canonical branch; retain only changes not already included upstream, resolving any API differences.
Rebase this TRT-LLM PR onto latest main, including the refreshed compatibility patch and integration changes.
Keep vendors.flashinfer-prims-ts in 3rdparty/vendor_sources.lock.yaml pointing to your temporary source branch, update its immutable commit, and regenerate the selected vendored files and digests. Do not restore the old canonical branch/pin while resolving conflicts.
Update the PR description's vendor-promotion block if the paired upstream PRs or explicit commit resolutions change after rebasing.
Verify the vendored snapshot and rerun the relevant tests and CI.

The publishing bot now targets the new dated canonical branch. After this source-update PR merges, its lock-only promotion will target that branch. The old trtllm-prims-ts-dev branch is intentionally unchanged.

yuxianq · 2026-10-13T09:32:53Z

/bot run

tensorrt-cicd · 2026-10-13T09:41:04Z

PR_Github #77492 [ run ] triggered by Bot. Commit: 72a6cea Link to invocation

tensorrt-cicd · 2026-10-13T12:44:59Z

PR_Github #77492 [ run ] completed with state SUCCESS. Commit: 72a6cea
/LLM/main/L0_MergeRequest_PR pipeline #63880 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-10-22T10:57:18Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

The pinned temporary source commit e377847b798c8203eec83802ea81794bf07303f1 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip (observed ae3851ab5f958ba22f154059df5b38321f6369d2). Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

yuxianq · 2026-10-23T05:38:08Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/heyuhhh/flashinfer.git, branch trtllm-prims-ts-dev.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

yuxianq · 2026-10-23T06:34:43Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip without merge commits, not just the consumer lock's pin 0e0761ef5ef10cb0762cc1b890a5c1a329bb49bf. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

yuxianq · 2026-10-26T07:37:38Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yunruis/flashinfer.git, branch trtllm-prims-ts-dev-20261023.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

yuxianq · 2026-10-26T10:19:39Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip without merge commits, not just the consumer lock's pin aa709061230a58858375c513056e7733fa74294a. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: 72a6ceab79ef. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin e377847b798c8203eec83802ea81794bf07303f1.

Supersedes previous vendor validation.

yuxianq · 2026-10-26T10:46:27Z

/bot run

yuxianq · 2026-10-26T10:47:18Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

No exact match was found for the following commits; similarity matching accepted them:

9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Checked PR revision: d829f61b3628. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 75961f74dac2c438cd7bce3dd135d527e78e49ec.

Supersedes previous vendor validation.

github-actions · 2026-10-26T10:52:49Z

👎 Promotion blocked, new vulnerability found

Vulnerability report

Component	Vulnerability	Description	Severity
py-filelock	CVE-2026-112906	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock versions from 4.1.0 can reuse a stale observation when an attacker pre-dates and removes its own marker, at least two lock-user processes are waiting, coarse or matching modification times preserve the observation, and an honest contender creates a fresh marker on the reused inode, causing that live marker to be removed as stale. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112914	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, SoftFileLock can restore a stale marker by name after moving it aside, allowing another local user with rename access in a non-sticky lock directory to swap in a symbolic link and, on a runtime that follows the link and a same-filesystem target, make the lock user hard-link one of its files into the lock directory. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112937	filelock is a platform-independent file lock for Python. From 4.1.0 until 4.2.1, ReadWriteLock can remove a randomized .filelock-* private-hard-link directory during cleanup, allowing another local user with rename access in a non-sticky parent to observe a live name or forge a dead-owner sweep name and rename a sibling directory onto it for deletion. This issue is fixed in version 4.2.1.	MEDIUM
py-filelock	CVE-2026-112940	filelock is a platform-independent file lock for Python. From 3.20.4 until 4.2.1, ReadWriteLock let SQLite reopen the database and side files by name; on POSIX systems an attacker with rename access in a non-sticky parent can swap those names to cause overwrite or unlink operations, while a writable adjacent hot journal is additionally required for attacker-chosen content overwrites. This issue is fixed in version 4.2.1.	HIGH

yuxianq · 2026-11-01T10:59:37Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

df3434b1dc23df96527e202aa43ae233ed8641ee: no exact match; best similarities: upstream PR #5489: 12.95%; no PR meets the 90.00% similarity threshold
06e9cc5c084a0f5791ae4989a9bda4a7a595eaca: no exact match; best similarities: upstream PR #5489: 2.63%; no PR meets the 90.00% similarity threshold
502353da9e0f22d8eb1d772ecebd830d08eafbfe: no exact match; best similarities: upstream PR #5489: 6.59%; no PR meets the 90.00% similarity threshold
9e26388ea95e5b40c924ffd2c6acef575cb5a8b1: no exact match; best similarities: upstream PR #5489: 7.84%; no PR meets the 90.00% similarity threshold
76e8229748606d0e51f27f84fe19335cbe605ae9: no exact match; best similarities: upstream PR #5489: 5.45%; no PR meets the 90.00% similarity threshold

No exact match was found for the following commits; similarity matching accepted them:

9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 0c80a0cfc644. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 76e8229748606d0e51f27f84fe19335cbe605ae9.

Supersedes previous vendor validation.

yuxianq · 2026-11-01T11:00:19Z

/bot run --extra-stage "GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1"

yuxianq · 2026-11-01T11:04:55Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

Upstream attribution is unresolved for these source commits:

df3434b1dc23df96527e202aa43ae233ed8641ee: no exact match; best similarities: upstream PR #5489: 12.95%, upstream PR #6774: 6.24%; no PR meets the 90.00% similarity threshold
06e9cc5c084a0f5791ae4989a9bda4a7a595eaca: no exact match; best similarities: upstream PR #6774: 3.92%, upstream PR #5489: 2.63%; no PR meets the 90.00% similarity threshold
9e26388ea95e5b40c924ffd2c6acef575cb5a8b1: no exact match; best similarities: upstream PR #6774: 10.39%, upstream PR #5489: 7.84%; no PR meets the 90.00% similarity threshold
76e8229748606d0e51f27f84fe19335cbe605ae9: no exact match; best similarities: upstream PR #6774: 9.41%, upstream PR #5489: 5.45%; no PR meets the 90.00% similarity threshold

No exact match was found for the following commits; similarity matching accepted them:

9b4e7c9357a3d865dea0e19fbc38916c55dd3de5: upstream PR #5489, similarity 92.06% — accepted (threshold 90.00%; similar-edits).

Scores compare normalized patch lines in lock-selected files, not semantic equivalence. These changes retain retain-until-verified refresh policy.

Fix: In the PR description, add missing PRs to vendors.flashinfer-prims-ts.upstream_prs, remove incorrectly listed PRs, or correct/rebase the source pin and update the vendored snapshot. For an intentional adaptation or ambiguous ownership, merge a targeted resolutions entry into the existing vendor metadata as shown below. Replace the SHA and URL and give a specific reason; preserve other entries. No complete commit_map is required. A resolution records an author assertion, not verified equivalence; it is retained during refresh until separately verified.

vendors:
  flashinfer-prims-ts:
    upstream_prs:
      - https://github.com/flashinfer-ai/flashinfer/pull/123
    resolutions:
      FULL_40_CHARACTER_SOURCE_SHA:
        upstream_pr: https://github.com/flashinfer-ai/flashinfer/pull/123
        reason: Explain the downstream adaptation or why this PR owns the change.

The bot will recheck automatically on a later poll.

Checked PR revision: 0c80a0cfc644. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 76e8229748606d0e51f27f84fe19335cbe605ae9.

Supersedes previous vendor validation.

tensorrt-cicd · 2026-11-01T11:07:00Z

PR_Github #81839 [ run ] triggered by Bot. Commit: 0c80a0c Link to invocation

tensorrt-cicd · 2026-11-01T13:59:30Z

PR_Github #81839 [ run ] completed with state SUCCESS. Commit: 0c80a0c
/LLM/main/L0_MergeRequest_PR pipeline #67491 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-11-04T21:35:18Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/harrisonzhy/flashinfer.git, branch harrisonzhy/trtllm-prims-ts-dev_vc-attn-20261029.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: a7da965b6118. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

yuxianq · 2026-11-05T07:39:36Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-ts-mla-prefill--flashinfer onto its latest tip without merge commits, not just the consumer lock's pin 7087b06c22ceb116b2236a938038a00b39f752d1. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: a7da965b6118. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-ts-mla-prefill--flashinfer, pin 069139a65aad98f3e5c61f049c761e28eff0b14e.

Supersedes previous vendor validation.

yuxianq · 2026-11-09T18:08:56Z

Vendor validation: `flashinfer-prims-ts`

@yuxianq — Vendor promotion needs maintainer action:

The base lock now uses refresh branch trtllm-prims-ts-dev-20261107, but this monitor still publishes to trtllm-prims-ts-dev-20261012.

Fix: A maintainer must stop this monitor and migrate its workdir with --canonical-branch trtllm-prims-ts-dev-20261107 --migrate-canonical-branch-from trtllm-prims-ts-dev-20261012. Verify a dry run before resuming publication. Do not promote back to the old branch or change contributor metadata to bypass this cutover.

The bot will recheck automatically on a later poll.

Checked PR revision: a7da965b6118. This is a snapshot; newer results are posted as new comments. Observed dev/upstream tip SHAs may be stale: tip movement alone does not post a new comment when the result is unchanged. Fetch the live branch before rebasing or choosing a new pin.

Supersedes previous vendor validation.

yuxianq · 2026-11-10T12:03:27Z

/bot run --disable-fail-fast --extra-stage "GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1"

tensorrt-cicd · 2026-11-10T12:11:14Z

PR_Github #83789 [ run ] triggered by Bot. Commit: 0660910 Link to invocation

…patch Add mla_context_preprocess with caller-owned buffers. Reuse RoPE/cache append and FP8 QKV quantization, allow preprocessed cached/chunked inputs, and pack the strided V projection. This commit adds the prologue contract only; the PrimTS adapter follows separately. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Route fresh, cached and chunked MLA prefill through the planned PrimTS context wrapper, including FP8 scales and merge-compatible softmax statistics. Preserve full cached K/V spans and account for BF16 V-packing workspace. Fold the FMHA-field and statistics-API rebase adaptations into this implementation. Correct MLA generation to require query-only input (is_fused_qkv=False), matching PhasedFmha. Include selector/contract and workspace tests; Rubin enablement and performance scheduling are separate commits. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Add K3 full-attention coverage for fresh/cached/chunked prefill, BF16/FP8 KV, TP/attention-DP heads, and CUDA graph decode. Check fresh prefill against independent FP64 math after preprocessing; retain direct backend comparisons for cached/chunk-merged prefill and decode. Add forced-PrimTS GSM8K CI cases, supported KvCacheConfig dtype selection, and the missing FP8-KV reference without relaxing accuracy thresholds. Hecate qualification on the equivalent full functional snapshot passed 1319 examples per dtype (BF16 96.058%, FP8 96.740%). Scheduling optimization is not part of this commit. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Keep Blackwell on the existing DSL 4.7 minimum and require DSL 4.8 for SM107, retaining the compiler 13.3 gate. Extend availability tests and full-attention test collection to Rubin. This is architecture enablement only, not the dense-Q scheduling optimization. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Permit explicit MLA context calls under linear speculative metadata, retaining tree, custom-mask, mixed-phase and Helix-context exclusions. Cover causal fresh/final chunks, FULL cached chunks, FP8 statistics and live metadata reuse. Validation: 363 adapter tests and 8 Kimi K3 module tests passed with no skips in the matching Rubin runtime; two independent review passes and pre-commit passed. Full-model MTP qualification remains separate. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Add the FP8 MLA DSpark7 integration case to the existing eight-GPU CI stage. Propagate strict target FMHA selection to pre-existing workers, restore each worker environment, and cover success and failure cleanup with CPU regressions. Preserve existing model resources, GSM8K criteria, acceptance length, and FP32 KDA state. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Preserve the static PrimTS capability contract after rebasing onto the shared MLA routing helper. SM107 FP8 pages are not exclusive to CuTe DSL once PrimTS supports them. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Avoid repeated workspace policy and sizing work on warmed eager MLA calls. Scope scalar sizes to the prepared KV capacity, invalidate dependent plans when that capacity changes, and preserve workspace allocation and CUDA graph capture checks. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Forward the trailing disabled KV-bounds operand through the downstream prepared launcher and regenerate the vendor compatibility patch with the vendor toolkit. Cover host argument parity for both cache layouts and workspace modes, and exercise prepared execution after persistent-kernel reload. Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

…ging Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

…shot Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

tensorrt-cicd · 2026-11-10T17:18:36Z

PR_Github #83789 [ run ] completed with state SUCCESS. Commit: 0660910
/LLM/main/L0_MergeRequest_PR pipeline #69283 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

yuxianq · 2026-11-10T17:38:40Z

/bot run --disable-fail-fast --extra-stage "GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1"

tensorrt-cicd · 2026-11-10T17:46:03Z

PR_Github #83851 [ run ] triggered by Bot. Commit: 6df6036 Link to invocation

github-actions Bot assigned yuxianq Sep 18, 2026

yuxianq mentioned this pull request Sep 23, 2026

feat(prims-ts): expose merge-compatible softmax statistics for prefill and decode flashinfer-ai/flashinfer#5489

Open

3 of 6 tasks

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 47a02ee to d575c3f Compare October 9, 2026 08:17

yuxianq mentioned this pull request Oct 12, 2026

[None][chore] Refresh PrimTS vendor source onto FlashInfer main #20155

Merged

1 task done

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 8298aba to 72a6cea Compare October 13, 2026 09:32

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 72a6cea to d829f61 Compare October 26, 2026 10:46

yuxianq mentioned this pull request Oct 27, 2026

[None][fix] Exclude vulnerable filelock releases #21071

Merged

5 tasks done

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from d829f61 to 0c80a0c Compare November 1, 2026 10:58

yuxianq changed the title ~~[None][feat] Add PrimTS MLA prefill and Kimi K3 coverage~~ [None][feat] Add PrimTS MLA prefill, MTP, and Rubin support Nov 1, 2026

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from a7da965 to 0660910 Compare November 10, 2026 12:00

yuxianq added 20 commits November 10, 2026 16:29

[None][feat] Integrate linear MTP and Helix bounds in PrimTS MLA

7d1c4c4

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][test] Cover FP8 MLA linear MTP with real K3 modules

d23ad8d

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][perf] Normalize dense MLA context capacity buckets

1ab5362

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][perf] Prewarm selected dense MLA profiles before readiness

ba5dd56

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][test] Initialize PrimTS warmup collaborators in phase fixtures

a074ee7

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][fix] Preserve original softmax scales for PrimTS partition mer…

d67af79

…ging Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][feat] Broaden PrimTS MLA context geometries

373f3ec

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][perf] Avoid non-FP8 MLA value packing in PrimTS

f7a0200

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][chore] Rebase PrimTS vendor onto the 20261107 development snap…

3aa9939

…shot Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][fix] Adapt MLA integration and warmup to refreshed PrimTS APIs

d745d03

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

[None][test] Consolidate MLA preprocessing coverage

6df6036

Signed-off-by: Yuxian Qiu <142763828+yuxianq@users.noreply.github.com>

yuxianq force-pushed the feat/prims-ts-mla-prefill branch from 0660910 to 6df6036 Compare November 10, 2026 17:38

Conversation

yuxianq commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Review order

Vendored source

Latest main rebase validation

Previous source-refresh validation

Historical performance and accuracy — unchanged, not refreshed-head qualification

Main-based prepared-context comparison on Rubin

Functionality-only Kimi K3 qualification on Rubin

Controlled DEP16 performance experiment on Rubin

Accuracy and scope limits

PR Checklist

GitHub Bot Help

Uh oh!

yuxianq commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuxianq commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

yuxianq commented Oct 13, 2026

Uh oh!

yuxianq commented Oct 13, 2026

Uh oh!

tensorrt-cicd commented Oct 13, 2026

Uh oh!

tensorrt-cicd commented Oct 13, 2026

Uh oh!

yuxianq commented Oct 22, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Oct 23, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Oct 23, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Oct 26, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Oct 26, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Oct 26, 2026

Uh oh!

yuxianq commented Oct 26, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

github-actions Bot commented Oct 26, 2026

👎 Promotion blocked, new vulnerability found

Vulnerability report

Uh oh!

yuxianq commented Nov 1, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Nov 1, 2026

Uh oh!

yuxianq commented Nov 1, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

tensorrt-cicd commented Nov 1, 2026

Uh oh!

yuxianq commented Nov 4, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Nov 5, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Nov 9, 2026

Vendor validation: flashinfer-prims-ts

Uh oh!

yuxianq commented Nov 10, 2026

Uh oh!

yuxianq commented Sep 18, 2026 •

edited

Loading

yuxianq commented Oct 9, 2026 •

edited

Loading

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`

Vendor validation: `flashinfer-prims-ts`